Papers with bias measurement metrics
IndiBias: A Benchmark Dataset to Measure Social Biases in Language Models for Indian Context (2024.naacl-long)
Copied to clipboard
Nihar Sahoo, Pranamya Kulkarni, Arif Ahmad, Tanu Goyal, Narjis Asad, Aparna Garimella, Pushpak Bhattacharyya
| Challenge: | Existing benchmark datasets focus on English language and the Western context, leaving a void for a reliable dataset that encapsulates India’s unique socio-cultural nuances. |
| Approach: | They propose to use CrowS-Pairs to create a benchmark dataset that captures and evaluates social biases in Large Language Models (LLMs). |
| Outcome: | The proposed dataset is available in English and Hindi and leverages LLMs ChatGPT and InstructGPT to augment the existing dataset with diverse societal biases and stereotypes prevalent in India. |
Unpacking Bias: An Empirical Study of Bias Measurement Metrics, Mitigation Algorithms, and Their Interactions (2024.lrec-main)
Copied to clipboard
| Challenge: | Word embeddings (WE) models reflect gender, racial, and religious stereotypes from the corpus on which they are trained. |
| Approach: | They propose a method that carefully controls for word sets and vector normalization to address these factors. |
| Outcome: | The proposed method detects consistency between different mitigation methods and the evaluation words used by the mitigation methods. |